notebook.community

Edit and run 





In [7]:



    


import sys
sys.path.append('../deeprl')
import gym
import tensorflow as tf
import tflearn as nn
import numpy as np
#create the environment
env = gym.make('CartPole-v1')



    


















    






[2017-02-26 19:20:48,884] Making new env: CartPole-v1

First we create a policy network. This network takes an observation as input and returns action porbabilities.





In [2]:



    


def create_mlp(inputs, n_out):
    net = nn.input_data(placeholder=inputs)
    net = nn.fully_connected(net, 25, activation='relu', weight_decay=0.0001)
    net = nn.dropout(net, 0.3)
    net = nn.fully_connected(net, 25, activation='relu', weight_decay=0.0001)
    net = nn.dropout(net, 0.3)
    #softmax to normalize to probabilities
    net = nn.fully_connected(net, n_out, activation='softmax')
    return net

We can now apply this network to the environment. On every step the network produces action probabilities for the current observation that can be used to select actions





In [12]:



    


# create a policy network
nn_input = tf.placeholder("float",(None,)+env.observation_space.shape)
n_acts = env.action_space.n
policy = create_mlp(nn_input, env.action_space.n)

# run the network for a single episode
obs = env.reset()
done = False
session = tf.Session()
session.run(tf.global_variables_initializer())
steps = 0
while not done:
    # get action probs for current observation
    act_probs = policy.eval(session=session,feed_dict={nn_input:obs.reshape(1,-1)}).flatten()
    # select action according to probs
    act = np.random.choice(n_acts, p = act_probs)
    #step environment
    obs,rew,done,_ = env.step(act)
    print('step % d reward %f'%(steps,rew))
    steps += 1



    


















    






step  0 reward 1.000000
step  1 reward 1.000000
step  2 reward 1.000000
step  3 reward 1.000000
step  4 reward 1.000000
step  5 reward 1.000000
step  6 reward 1.000000
step  7 reward 1.000000
step  8 reward 1.000000
step  9 reward 1.000000
step  10 reward 1.000000
step  11 reward 1.000000
step  12 reward 1.000000
step  13 reward 1.000000
step  14 reward 1.000000

As the network uses randomely initialized weights, we cannot expect the resulting policy to perform well. To goal of learning now becomes to adapt the network weights such that the performance of the policy improves. In order to do this, we first define a cost function that we want to optimize:

$$ J(\theta) = E_{\pi_\theta}[R] = \sum_\tau p(\tau \mid \pi_\theta)R(\tau) $$

This cost function indicates we want to optimize the expected return (sum of discounted rewards) for the policy. Here $\pi_\theta$ is the policy represented by the network with weights $\theta$, $p(\tau \mid \pi_\theta)$ is the probability of observing a trajectory (episode) $\tau= s_0,a_0,s_1,a_t,...,s_T$ when applying this policy and $R(\tau)$ is the return for this trajectory, ie $R = \sum_{t=0}^T \gamma^t r_{t+1}$. Changing our policy changes the probability $p$ of achieving a trajectory, so essentially this cost says that we want to change the policy in such a way that the probability of trajectories which have a high return increases.

In order to optimize this cost function, we calculate the gradient w.r.t. the network weights:

$$\nabla_\theta J(\theta) = \nabla_\theta \sum_\tau p(\tau \mid \pi_\theta)R(\tau) = \sum_\tau \nabla_\theta p(\tau \mid \pi_\theta)R(\tau)$$

Using $\nabla_\theta p(\tau \mid \pi_\theta) = \frac{p(\tau \mid \pi_\theta)}{p(\tau \mid \pi_\theta)} \nabla_\theta p(\tau \mid \pi_\theta)$ and $\nabla_\theta log p(\tau \mid \pi_\theta) = \frac{1}{p(\tau \mid \pi_\theta)}\nabla_\theta p(\tau \mid \pi_\theta)$ we can rewrite this as:

$$ \sum_\tau p(\tau \mid \pi_\theta) \nabla_\theta log p(\tau \mid \pi_\theta) R(\tau)$$$$ = E_{\pi_\theta}[ \nabla_\theta log p(\tau \mid \pi_\theta) R(\tau) ] $$

Where $E_{\pi_\theta}$ is an expectation over trajectories generated by the policy. This means we can sample this expectation by running the policy and averaging over the generated trajectories.

Moreover, we do not actually need to compute the the probability $p(\tau \mid \pi_\theta)$ of a trajectory. This probability of $\tau= s_0,a_0,s_1,a_t,...,s_T$ is $\prod _{t=0}^T T(s_t,a_t,s_{t+1}) \pi_\theta(s_t,a_t)$, where $T(s_0,a_0,s_1)$ are the transition probabilities of the MDP. Since $T$ does not depend on the network weights, we only have to look at $\pi$ to compute the gradient $ \nabla_\theta log p(\tau \mid \pi_\theta)$:

$$\nabla_\theta log p(\tau \mid \pi_\theta)= \nabla_\theta log \prod _{t=0}^T \pi_\theta(s_t,a_t)T(s_t,a_t,s_{t+1}) \\ = \sum_{t=0}^T \nabla_\theta log \pi_\theta(s_t,a_t) $$

This results in the final formula $\nabla_\theta J(\theta) = E_{\pi_\theta}[ \sum_{t=0}^T \nabla_\theta log \pi_\theta(s_t,a_t) R(\tau) ]$, i.e. to compute the policy gradient from a trajectory, we sum over the log of the action probabilities output by the network and multiply by the return.

This results in the Monte Carlo Policy gradient (REINFORCE) algorithm:

In tensorflow this can be implemented by defining the following loss:

loss_op = -tf.reduce_mean(tf.log(probs) * R)

where probs is the output of the network for the selected actions for each time step and R is the return (here: sum of discounted rewards) starting from that time step. Here we take a mean over time steps rather than a sum. This helps with variations in magnitude because trajectories can have very different lengths.

You can run the full policy gradient algorithm using the DPGAgent:





In [14]:



    


from agents import DPGAgent
from experiment import Experiment

#erase previously created tf graphs
tf.reset_default_graph()

#create a DQNAgent
agent = DPGAgent(create_mlp, env.action_space.n, env.observation_space.shape, alpha=0.001, gamma=0.99)
# create experiment on cartpole
exp = Experiment(agent, env)
#run agent for 150K steps and return episode rewards
rew = exp.run_epoch(150000)



    


















    






WARNING:tensorflow:From ../deeprl/agents.py:224: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.










    






[2017-02-26 20:18:37,257] From ../deeprl/agents.py:224: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.










    






episode reward: 15.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 29.000000
episode reward: 14.000000
episode reward: 35.000000
episode reward: 11.000000
episode reward: 9.000000
episode reward: 12.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 20.000000
episode reward: 33.000000
episode reward: 17.000000
episode reward: 32.000000
episode reward: 26.000000
episode reward: 20.000000
episode reward: 22.000000
episode reward: 38.000000
episode reward: 25.000000
episode reward: 11.000000
episode reward: 14.000000
episode reward: 10.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 63.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 17.000000
episode reward: 25.000000
episode reward: 45.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 21.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 28.000000
episode reward: 14.000000
episode reward: 10.000000
episode reward: 22.000000
episode reward: 55.000000
episode reward: 88.000000
episode reward: 10.000000
episode reward: 14.000000
episode reward: 34.000000
episode reward: 26.000000
episode reward: 13.000000
episode reward: 41.000000
episode reward: 25.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 12.000000
episode reward: 41.000000
episode reward: 30.000000
episode reward: 14.000000
episode reward: 15.000000
episode reward: 32.000000
episode reward: 52.000000
episode reward: 13.000000
episode reward: 33.000000
episode reward: 10.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 48.000000
episode reward: 19.000000
episode reward: 51.000000
episode reward: 43.000000
episode reward: 31.000000
episode reward: 15.000000
episode reward: 23.000000
episode reward: 20.000000
episode reward: 51.000000
episode reward: 30.000000
episode reward: 27.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 49.000000
episode reward: 28.000000
episode reward: 11.000000
episode reward: 34.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 28.000000
episode reward: 16.000000
episode reward: 18.000000
episode reward: 22.000000
episode reward: 10.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 59.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 14.000000
episode reward: 21.000000
episode reward: 29.000000
episode reward: 26.000000
episode reward: 23.000000
episode reward: 10.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 23.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 15.000000
episode reward: 37.000000
episode reward: 21.000000
episode reward: 27.000000
episode reward: 11.000000
episode reward: 23.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 15.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 36.000000
episode reward: 18.000000
episode reward: 40.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 59.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 9.000000
episode reward: 18.000000
episode reward: 20.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 37.000000
episode reward: 27.000000
episode reward: 33.000000
episode reward: 27.000000
episode reward: 12.000000
episode reward: 9.000000
episode reward: 21.000000
episode reward: 9.000000
episode reward: 16.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 10.000000
episode reward: 11.000000
episode reward: 10.000000
episode reward: 64.000000
episode reward: 23.000000
episode reward: 75.000000
episode reward: 13.000000
episode reward: 24.000000
episode reward: 29.000000
episode reward: 21.000000
episode reward: 25.000000
episode reward: 20.000000
episode reward: 10.000000
episode reward: 19.000000
episode reward: 16.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 10.000000
episode reward: 30.000000
episode reward: 16.000000
episode reward: 22.000000
episode reward: 9.000000
episode reward: 10.000000
episode reward: 34.000000
episode reward: 17.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 76.000000
episode reward: 14.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 9.000000
episode reward: 13.000000
episode reward: 33.000000
episode reward: 14.000000
episode reward: 12.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 26.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 27.000000
episode reward: 51.000000
episode reward: 33.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 14.000000
episode reward: 26.000000
episode reward: 10.000000
episode reward: 39.000000
episode reward: 11.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 18.000000
episode reward: 12.000000
episode reward: 21.000000
episode reward: 21.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 15.000000
episode reward: 18.000000
episode reward: 34.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 15.000000
episode reward: 23.000000
episode reward: 23.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 22.000000
episode reward: 21.000000
episode reward: 10.000000
episode reward: 34.000000
episode reward: 13.000000
episode reward: 10.000000
episode reward: 46.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 31.000000
episode reward: 11.000000
episode reward: 34.000000
episode reward: 23.000000
episode reward: 30.000000
episode reward: 24.000000
episode reward: 15.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 36.000000
episode reward: 16.000000
episode reward: 13.000000
episode reward: 23.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 33.000000
episode reward: 39.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 29.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 16.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 19.000000
episode reward: 24.000000
episode reward: 17.000000
episode reward: 11.000000
episode reward: 22.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 12.000000
episode reward: 21.000000
episode reward: 11.000000
episode reward: 47.000000
episode reward: 20.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 35.000000
episode reward: 50.000000
episode reward: 35.000000
episode reward: 14.000000
episode reward: 17.000000
episode reward: 21.000000
episode reward: 36.000000
episode reward: 21.000000
episode reward: 29.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 37.000000
episode reward: 55.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 24.000000
episode reward: 10.000000
episode reward: 37.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 68.000000
episode reward: 15.000000
episode reward: 25.000000
episode reward: 11.000000
episode reward: 18.000000
episode reward: 20.000000
episode reward: 41.000000
episode reward: 23.000000
episode reward: 38.000000
episode reward: 32.000000
episode reward: 17.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 16.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 17.000000
episode reward: 37.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 43.000000
episode reward: 21.000000
episode reward: 9.000000
episode reward: 36.000000
episode reward: 32.000000
episode reward: 17.000000
episode reward: 60.000000
episode reward: 24.000000
episode reward: 27.000000
episode reward: 14.000000
episode reward: 30.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 49.000000
episode reward: 15.000000
episode reward: 26.000000
episode reward: 11.000000
episode reward: 43.000000
episode reward: 23.000000
episode reward: 45.000000
episode reward: 31.000000
episode reward: 15.000000
episode reward: 20.000000
episode reward: 28.000000
episode reward: 29.000000
episode reward: 39.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 12.000000
episode reward: 64.000000
episode reward: 43.000000
episode reward: 44.000000
episode reward: 14.000000
episode reward: 9.000000
episode reward: 14.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 24.000000
episode reward: 39.000000
episode reward: 79.000000
episode reward: 15.000000
episode reward: 11.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 32.000000
episode reward: 17.000000
episode reward: 11.000000
episode reward: 17.000000
episode reward: 26.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 52.000000
episode reward: 20.000000
episode reward: 66.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 57.000000
episode reward: 42.000000
episode reward: 33.000000
episode reward: 18.000000
episode reward: 24.000000
episode reward: 13.000000
episode reward: 31.000000
episode reward: 37.000000
episode reward: 36.000000
episode reward: 27.000000
episode reward: 30.000000
episode reward: 30.000000
episode reward: 22.000000
episode reward: 35.000000
episode reward: 61.000000
episode reward: 44.000000
episode reward: 10.000000
episode reward: 23.000000
episode reward: 16.000000
episode reward: 23.000000
episode reward: 11.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 45.000000
episode reward: 21.000000
episode reward: 23.000000
episode reward: 21.000000
episode reward: 21.000000
episode reward: 32.000000
episode reward: 79.000000
episode reward: 50.000000
episode reward: 30.000000
episode reward: 17.000000
episode reward: 17.000000
episode reward: 41.000000
episode reward: 41.000000
episode reward: 31.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 22.000000
episode reward: 16.000000
episode reward: 68.000000
episode reward: 20.000000
episode reward: 33.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 44.000000
episode reward: 16.000000
episode reward: 72.000000
episode reward: 12.000000
episode reward: 28.000000
episode reward: 23.000000
episode reward: 88.000000
episode reward: 10.000000
episode reward: 24.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 28.000000
episode reward: 12.000000
episode reward: 15.000000
episode reward: 33.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 47.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 29.000000
episode reward: 35.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 18.000000
episode reward: 23.000000
episode reward: 29.000000
episode reward: 12.000000
episode reward: 29.000000
episode reward: 13.000000
episode reward: 21.000000
episode reward: 26.000000
episode reward: 21.000000
episode reward: 19.000000
episode reward: 32.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 23.000000
episode reward: 17.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 29.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 11.000000
episode reward: 38.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 16.000000
episode reward: 39.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 14.000000
episode reward: 18.000000
episode reward: 28.000000
episode reward: 14.000000
episode reward: 17.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 11.000000
episode reward: 25.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 47.000000
episode reward: 31.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 31.000000
episode reward: 13.000000
episode reward: 21.000000
episode reward: 27.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 12.000000
episode reward: 10.000000
episode reward: 13.000000
episode reward: 25.000000
episode reward: 17.000000
episode reward: 32.000000
episode reward: 21.000000
episode reward: 28.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 21.000000
episode reward: 35.000000
episode reward: 9.000000
episode reward: 20.000000
episode reward: 25.000000
episode reward: 9.000000
episode reward: 20.000000
episode reward: 13.000000
episode reward: 11.000000
episode reward: 26.000000
episode reward: 35.000000
episode reward: 26.000000
episode reward: 21.000000
episode reward: 30.000000
episode reward: 17.000000
episode reward: 25.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 23.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 17.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 39.000000
episode reward: 9.000000
episode reward: 42.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 18.000000
episode reward: 28.000000
episode reward: 17.000000
episode reward: 21.000000
episode reward: 16.000000
episode reward: 22.000000
episode reward: 31.000000
episode reward: 22.000000
episode reward: 27.000000
episode reward: 18.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 36.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 36.000000
episode reward: 24.000000
episode reward: 28.000000
episode reward: 34.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 40.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 16.000000
episode reward: 27.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 12.000000
episode reward: 26.000000
episode reward: 31.000000
episode reward: 24.000000
episode reward: 80.000000
episode reward: 17.000000
episode reward: 40.000000
episode reward: 28.000000
episode reward: 19.000000
episode reward: 16.000000
episode reward: 23.000000
episode reward: 10.000000
episode reward: 24.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 10.000000
episode reward: 29.000000
episode reward: 22.000000
episode reward: 24.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 24.000000
episode reward: 28.000000
episode reward: 65.000000
episode reward: 13.000000
episode reward: 14.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 25.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 14.000000
episode reward: 24.000000
episode reward: 23.000000
episode reward: 27.000000
episode reward: 23.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 23.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 25.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 29.000000
episode reward: 19.000000
episode reward: 94.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 28.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 36.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 30.000000
episode reward: 23.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 12.000000
episode reward: 29.000000
episode reward: 23.000000
episode reward: 30.000000
episode reward: 34.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 22.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 82.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 50.000000
episode reward: 22.000000
episode reward: 13.000000
episode reward: 18.000000
episode reward: 10.000000
episode reward: 19.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 33.000000
episode reward: 33.000000
episode reward: 27.000000
episode reward: 13.000000
episode reward: 29.000000
episode reward: 30.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 13.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 32.000000
episode reward: 25.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 11.000000
episode reward: 13.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 11.000000
episode reward: 39.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 21.000000
episode reward: 33.000000
episode reward: 10.000000
episode reward: 30.000000
episode reward: 17.000000
episode reward: 17.000000
episode reward: 24.000000
episode reward: 18.000000
episode reward: 19.000000
episode reward: 18.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 25.000000
episode reward: 59.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 12.000000
episode reward: 11.000000
episode reward: 12.000000
episode reward: 32.000000
episode reward: 23.000000
episode reward: 33.000000
episode reward: 21.000000
episode reward: 11.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 23.000000
episode reward: 55.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 25.000000
episode reward: 21.000000
episode reward: 18.000000
episode reward: 22.000000
episode reward: 19.000000
episode reward: 49.000000
episode reward: 11.000000
episode reward: 27.000000
episode reward: 23.000000
episode reward: 12.000000
episode reward: 22.000000
episode reward: 14.000000
episode reward: 36.000000
episode reward: 35.000000
episode reward: 14.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 24.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 11.000000
episode reward: 42.000000
episode reward: 21.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 33.000000
episode reward: 26.000000
episode reward: 22.000000
episode reward: 12.000000
episode reward: 12.000000
episode reward: 43.000000
episode reward: 46.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 73.000000
episode reward: 28.000000
episode reward: 18.000000
episode reward: 29.000000
episode reward: 22.000000
episode reward: 11.000000
episode reward: 40.000000
episode reward: 15.000000
episode reward: 37.000000
episode reward: 31.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 23.000000
episode reward: 35.000000
episode reward: 24.000000
episode reward: 72.000000
episode reward: 16.000000
episode reward: 33.000000
episode reward: 16.000000
episode reward: 59.000000
episode reward: 22.000000
episode reward: 14.000000
episode reward: 28.000000
episode reward: 28.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 44.000000
episode reward: 14.000000
episode reward: 36.000000
episode reward: 39.000000
episode reward: 28.000000
episode reward: 39.000000
episode reward: 23.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 23.000000
episode reward: 48.000000
episode reward: 10.000000
episode reward: 26.000000
episode reward: 14.000000
episode reward: 54.000000
episode reward: 38.000000
episode reward: 22.000000
episode reward: 34.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 35.000000
episode reward: 40.000000
episode reward: 55.000000
episode reward: 30.000000
episode reward: 31.000000
episode reward: 21.000000
episode reward: 30.000000
episode reward: 15.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 19.000000
episode reward: 44.000000
episode reward: 65.000000
episode reward: 14.000000
episode reward: 23.000000
episode reward: 20.000000
episode reward: 37.000000
episode reward: 25.000000
episode reward: 28.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 11.000000
episode reward: 52.000000
episode reward: 15.000000
episode reward: 11.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 40.000000
episode reward: 15.000000
episode reward: 49.000000
episode reward: 36.000000
episode reward: 37.000000
episode reward: 68.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 26.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 21.000000
episode reward: 37.000000
episode reward: 11.000000
episode reward: 21.000000
episode reward: 30.000000
episode reward: 46.000000
episode reward: 22.000000
episode reward: 22.000000
episode reward: 25.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 47.000000
episode reward: 57.000000
episode reward: 24.000000
episode reward: 51.000000
episode reward: 30.000000
episode reward: 61.000000
episode reward: 12.000000
episode reward: 35.000000
episode reward: 67.000000
episode reward: 34.000000
episode reward: 16.000000
episode reward: 45.000000
episode reward: 19.000000
episode reward: 9.000000
episode reward: 25.000000
episode reward: 23.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 24.000000
episode reward: 85.000000
episode reward: 18.000000
episode reward: 41.000000
episode reward: 55.000000
episode reward: 24.000000
episode reward: 44.000000
episode reward: 12.000000
episode reward: 24.000000
episode reward: 26.000000
episode reward: 14.000000
episode reward: 11.000000
episode reward: 13.000000
episode reward: 20.000000
episode reward: 19.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 81.000000
episode reward: 53.000000
episode reward: 29.000000
episode reward: 12.000000
episode reward: 17.000000
episode reward: 28.000000
episode reward: 71.000000
episode reward: 12.000000
episode reward: 45.000000
episode reward: 22.000000
episode reward: 32.000000
episode reward: 18.000000
episode reward: 23.000000
episode reward: 64.000000
episode reward: 17.000000
episode reward: 12.000000
episode reward: 26.000000
episode reward: 10.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 19.000000
episode reward: 25.000000
episode reward: 20.000000
episode reward: 14.000000
episode reward: 15.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 14.000000
episode reward: 39.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 34.000000
episode reward: 18.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 22.000000
episode reward: 12.000000
episode reward: 20.000000
episode reward: 44.000000
episode reward: 15.000000
episode reward: 58.000000
episode reward: 26.000000
episode reward: 77.000000
episode reward: 34.000000
episode reward: 41.000000
episode reward: 16.000000
episode reward: 27.000000
episode reward: 17.000000
episode reward: 22.000000
episode reward: 38.000000
episode reward: 28.000000
episode reward: 35.000000
episode reward: 21.000000
episode reward: 38.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 17.000000
episode reward: 14.000000
episode reward: 32.000000
episode reward: 13.000000
episode reward: 72.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 36.000000
episode reward: 24.000000
episode reward: 13.000000
episode reward: 9.000000
episode reward: 35.000000
episode reward: 25.000000
episode reward: 28.000000
episode reward: 56.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 21.000000
episode reward: 69.000000
episode reward: 18.000000
episode reward: 24.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 12.000000
episode reward: 11.000000
episode reward: 22.000000
episode reward: 21.000000
episode reward: 46.000000
episode reward: 22.000000
episode reward: 22.000000
episode reward: 49.000000
episode reward: 36.000000
episode reward: 82.000000
episode reward: 33.000000
episode reward: 14.000000
episode reward: 16.000000
episode reward: 31.000000
episode reward: 39.000000
episode reward: 76.000000
episode reward: 12.000000
episode reward: 46.000000
episode reward: 27.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 23.000000
episode reward: 32.000000
episode reward: 11.000000
episode reward: 33.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 32.000000
episode reward: 40.000000
episode reward: 41.000000
episode reward: 40.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 42.000000
episode reward: 15.000000
episode reward: 81.000000
episode reward: 40.000000
episode reward: 24.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 37.000000
episode reward: 22.000000
episode reward: 31.000000
episode reward: 25.000000
episode reward: 13.000000
episode reward: 64.000000
episode reward: 11.000000
episode reward: 41.000000
episode reward: 19.000000
episode reward: 10.000000
episode reward: 11.000000
episode reward: 26.000000
episode reward: 29.000000
episode reward: 15.000000
episode reward: 44.000000
episode reward: 16.000000
episode reward: 10.000000
episode reward: 37.000000
episode reward: 13.000000
episode reward: 39.000000
episode reward: 33.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 65.000000
episode reward: 52.000000
episode reward: 11.000000
episode reward: 24.000000
episode reward: 17.000000
episode reward: 26.000000
episode reward: 14.000000
episode reward: 17.000000
episode reward: 40.000000
episode reward: 23.000000
episode reward: 9.000000
episode reward: 15.000000
episode reward: 22.000000
episode reward: 36.000000
episode reward: 20.000000
episode reward: 36.000000
episode reward: 9.000000
episode reward: 9.000000
episode reward: 29.000000
episode reward: 62.000000
episode reward: 15.000000
episode reward: 21.000000
episode reward: 11.000000
episode reward: 17.000000
episode reward: 26.000000
episode reward: 18.000000
episode reward: 15.000000
episode reward: 58.000000
episode reward: 14.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 17.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 25.000000
episode reward: 11.000000
episode reward: 42.000000
episode reward: 46.000000
episode reward: 36.000000
episode reward: 19.000000
episode reward: 59.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 37.000000
episode reward: 16.000000
episode reward: 57.000000
episode reward: 65.000000
episode reward: 27.000000
episode reward: 22.000000
episode reward: 13.000000
episode reward: 29.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 12.000000
episode reward: 23.000000
episode reward: 28.000000
episode reward: 30.000000
episode reward: 18.000000
episode reward: 31.000000
episode reward: 29.000000
episode reward: 24.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 17.000000
episode reward: 42.000000
episode reward: 19.000000
episode reward: 11.000000
episode reward: 20.000000
episode reward: 39.000000
episode reward: 20.000000
episode reward: 19.000000
episode reward: 39.000000
episode reward: 24.000000
episode reward: 53.000000
episode reward: 28.000000
episode reward: 34.000000
episode reward: 25.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 29.000000
episode reward: 17.000000
episode reward: 55.000000
episode reward: 16.000000
episode reward: 59.000000
episode reward: 18.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 31.000000
episode reward: 84.000000
episode reward: 17.000000
episode reward: 15.000000
episode reward: 18.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 15.000000
episode reward: 36.000000
episode reward: 33.000000
episode reward: 22.000000
episode reward: 18.000000
episode reward: 33.000000
episode reward: 66.000000
episode reward: 44.000000
episode reward: 25.000000
episode reward: 70.000000
episode reward: 59.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 24.000000
episode reward: 20.000000
episode reward: 32.000000
episode reward: 21.000000
episode reward: 28.000000
episode reward: 57.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 35.000000
episode reward: 17.000000
episode reward: 117.000000
episode reward: 21.000000
episode reward: 45.000000
episode reward: 34.000000
episode reward: 100.000000
episode reward: 85.000000
episode reward: 31.000000
episode reward: 49.000000
episode reward: 28.000000
episode reward: 19.000000
episode reward: 41.000000
episode reward: 67.000000
episode reward: 22.000000
episode reward: 33.000000
episode reward: 27.000000
episode reward: 60.000000
episode reward: 23.000000
episode reward: 26.000000
episode reward: 35.000000
episode reward: 14.000000
episode reward: 85.000000
episode reward: 15.000000
episode reward: 41.000000
episode reward: 40.000000
episode reward: 22.000000
episode reward: 16.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 59.000000
episode reward: 65.000000
episode reward: 18.000000
episode reward: 26.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 53.000000
episode reward: 33.000000
episode reward: 66.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 38.000000
episode reward: 10.000000
episode reward: 25.000000
episode reward: 55.000000
episode reward: 26.000000
episode reward: 17.000000
episode reward: 30.000000
episode reward: 18.000000
episode reward: 28.000000
episode reward: 23.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 40.000000
episode reward: 11.000000
episode reward: 56.000000
episode reward: 34.000000
episode reward: 36.000000
episode reward: 51.000000
episode reward: 16.000000
episode reward: 23.000000
episode reward: 22.000000
episode reward: 23.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 10.000000
episode reward: 26.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 113.000000
episode reward: 81.000000
episode reward: 101.000000
episode reward: 25.000000
episode reward: 27.000000
episode reward: 28.000000
episode reward: 74.000000
episode reward: 34.000000
episode reward: 38.000000
episode reward: 54.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 32.000000
episode reward: 34.000000
episode reward: 43.000000
episode reward: 17.000000
episode reward: 35.000000
episode reward: 32.000000
episode reward: 24.000000
episode reward: 12.000000
episode reward: 75.000000
episode reward: 38.000000
episode reward: 46.000000
episode reward: 17.000000
episode reward: 14.000000
episode reward: 14.000000
episode reward: 38.000000
episode reward: 27.000000
episode reward: 15.000000
episode reward: 83.000000
episode reward: 17.000000
episode reward: 37.000000
episode reward: 32.000000
episode reward: 15.000000
episode reward: 20.000000
episode reward: 25.000000
episode reward: 40.000000
episode reward: 37.000000
episode reward: 20.000000
episode reward: 94.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 21.000000
episode reward: 14.000000
episode reward: 25.000000
episode reward: 31.000000
episode reward: 11.000000
episode reward: 23.000000
episode reward: 44.000000
episode reward: 22.000000
episode reward: 25.000000
episode reward: 15.000000
episode reward: 27.000000
episode reward: 14.000000
episode reward: 44.000000
episode reward: 51.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 87.000000
episode reward: 67.000000
episode reward: 28.000000
episode reward: 25.000000
episode reward: 18.000000
episode reward: 20.000000
episode reward: 44.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 35.000000
episode reward: 13.000000
episode reward: 35.000000
episode reward: 58.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 42.000000
episode reward: 30.000000
episode reward: 48.000000
episode reward: 13.000000
episode reward: 39.000000
episode reward: 87.000000
episode reward: 16.000000
episode reward: 62.000000
episode reward: 25.000000
episode reward: 39.000000
episode reward: 25.000000
episode reward: 25.000000
episode reward: 27.000000
episode reward: 23.000000
episode reward: 41.000000
episode reward: 34.000000
episode reward: 55.000000
episode reward: 21.000000
episode reward: 61.000000
episode reward: 17.000000
episode reward: 33.000000
episode reward: 29.000000
episode reward: 65.000000
episode reward: 54.000000
episode reward: 24.000000
episode reward: 37.000000
episode reward: 33.000000
episode reward: 70.000000
episode reward: 51.000000
episode reward: 32.000000
episode reward: 36.000000
episode reward: 49.000000
episode reward: 54.000000
episode reward: 11.000000
episode reward: 22.000000
episode reward: 29.000000
episode reward: 38.000000
episode reward: 107.000000
episode reward: 55.000000
episode reward: 34.000000
episode reward: 32.000000
episode reward: 29.000000
episode reward: 50.000000
episode reward: 17.000000
episode reward: 47.000000
episode reward: 30.000000
episode reward: 14.000000
episode reward: 26.000000
episode reward: 37.000000
episode reward: 21.000000
episode reward: 29.000000
episode reward: 42.000000
episode reward: 43.000000
episode reward: 46.000000
episode reward: 30.000000
episode reward: 19.000000
episode reward: 34.000000
episode reward: 19.000000
episode reward: 63.000000
episode reward: 28.000000
episode reward: 59.000000
episode reward: 77.000000
episode reward: 46.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 100.000000
episode reward: 24.000000
episode reward: 34.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 77.000000
episode reward: 36.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 22.000000
episode reward: 116.000000
episode reward: 14.000000
episode reward: 18.000000
episode reward: 51.000000
episode reward: 62.000000
episode reward: 25.000000
episode reward: 35.000000
episode reward: 16.000000
episode reward: 61.000000
episode reward: 43.000000
episode reward: 51.000000
episode reward: 10.000000
episode reward: 22.000000
episode reward: 18.000000
episode reward: 48.000000
episode reward: 22.000000
episode reward: 48.000000
episode reward: 62.000000
episode reward: 39.000000
episode reward: 62.000000
episode reward: 16.000000
episode reward: 70.000000
episode reward: 52.000000
episode reward: 12.000000
episode reward: 48.000000
episode reward: 43.000000
episode reward: 25.000000
episode reward: 53.000000
episode reward: 53.000000
episode reward: 50.000000
episode reward: 60.000000
episode reward: 30.000000
episode reward: 55.000000
episode reward: 43.000000
episode reward: 46.000000
episode reward: 15.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 19.000000
episode reward: 60.000000
episode reward: 9.000000
episode reward: 70.000000
episode reward: 140.000000
episode reward: 78.000000
episode reward: 35.000000
episode reward: 31.000000
episode reward: 28.000000
episode reward: 53.000000
episode reward: 44.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 39.000000
episode reward: 24.000000
episode reward: 44.000000
episode reward: 24.000000
episode reward: 24.000000
episode reward: 37.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 27.000000
episode reward: 123.000000
episode reward: 23.000000
episode reward: 33.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 40.000000
episode reward: 58.000000
episode reward: 44.000000
episode reward: 46.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 40.000000
episode reward: 83.000000
episode reward: 47.000000
episode reward: 31.000000
episode reward: 34.000000
episode reward: 27.000000
episode reward: 42.000000
episode reward: 37.000000
episode reward: 51.000000
episode reward: 18.000000
episode reward: 64.000000
episode reward: 35.000000
episode reward: 28.000000
episode reward: 73.000000
episode reward: 30.000000
episode reward: 62.000000
episode reward: 42.000000
episode reward: 74.000000
episode reward: 28.000000
episode reward: 55.000000
episode reward: 26.000000
episode reward: 31.000000
episode reward: 12.000000
episode reward: 72.000000
episode reward: 27.000000
episode reward: 33.000000
episode reward: 18.000000
episode reward: 75.000000
episode reward: 41.000000
episode reward: 38.000000
episode reward: 23.000000
episode reward: 84.000000
episode reward: 56.000000
episode reward: 90.000000
episode reward: 26.000000
episode reward: 54.000000
episode reward: 23.000000
episode reward: 49.000000
episode reward: 12.000000
episode reward: 37.000000
episode reward: 54.000000
episode reward: 57.000000
episode reward: 113.000000
episode reward: 50.000000
episode reward: 20.000000
episode reward: 86.000000
episode reward: 38.000000
episode reward: 61.000000
episode reward: 57.000000
episode reward: 91.000000
episode reward: 46.000000
episode reward: 27.000000
episode reward: 44.000000
episode reward: 64.000000
episode reward: 103.000000
episode reward: 28.000000
episode reward: 13.000000
episode reward: 38.000000
episode reward: 36.000000
episode reward: 84.000000
episode reward: 23.000000
episode reward: 82.000000
episode reward: 48.000000
episode reward: 48.000000
episode reward: 15.000000
episode reward: 111.000000
episode reward: 15.000000
episode reward: 26.000000
episode reward: 48.000000
episode reward: 47.000000
episode reward: 21.000000
episode reward: 107.000000
episode reward: 51.000000
episode reward: 34.000000
episode reward: 31.000000
episode reward: 9.000000
episode reward: 100.000000
episode reward: 74.000000
episode reward: 62.000000
episode reward: 11.000000
episode reward: 36.000000
episode reward: 45.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 59.000000
episode reward: 51.000000
episode reward: 26.000000
episode reward: 23.000000
episode reward: 73.000000
episode reward: 58.000000
episode reward: 29.000000
episode reward: 104.000000
episode reward: 15.000000
episode reward: 15.000000
episode reward: 32.000000
episode reward: 20.000000
episode reward: 82.000000
episode reward: 51.000000
episode reward: 28.000000
episode reward: 13.000000
episode reward: 28.000000
episode reward: 57.000000
episode reward: 28.000000
episode reward: 37.000000
episode reward: 57.000000
episode reward: 51.000000
episode reward: 26.000000
episode reward: 55.000000
episode reward: 39.000000
episode reward: 13.000000
episode reward: 99.000000
episode reward: 19.000000
episode reward: 41.000000
episode reward: 33.000000
episode reward: 36.000000
episode reward: 37.000000
episode reward: 40.000000
episode reward: 26.000000
episode reward: 40.000000
episode reward: 28.000000
episode reward: 17.000000
episode reward: 52.000000
episode reward: 39.000000
episode reward: 18.000000
episode reward: 66.000000
episode reward: 38.000000
episode reward: 81.000000
episode reward: 78.000000
episode reward: 59.000000
episode reward: 69.000000
episode reward: 23.000000
episode reward: 54.000000
episode reward: 22.000000
episode reward: 139.000000
episode reward: 28.000000
episode reward: 44.000000
episode reward: 86.000000
episode reward: 30.000000
episode reward: 34.000000
episode reward: 31.000000
episode reward: 48.000000
episode reward: 89.000000
episode reward: 42.000000
episode reward: 12.000000
episode reward: 46.000000
episode reward: 32.000000
episode reward: 63.000000
episode reward: 24.000000
episode reward: 23.000000
episode reward: 19.000000
episode reward: 29.000000
episode reward: 103.000000
episode reward: 36.000000
episode reward: 48.000000
episode reward: 36.000000
episode reward: 56.000000
episode reward: 47.000000
episode reward: 56.000000
episode reward: 115.000000
episode reward: 49.000000
episode reward: 62.000000
episode reward: 71.000000
episode reward: 38.000000
episode reward: 16.000000
episode reward: 18.000000
episode reward: 53.000000
episode reward: 16.000000
episode reward: 20.000000
episode reward: 13.000000
episode reward: 73.000000
episode reward: 30.000000
episode reward: 25.000000
episode reward: 22.000000
episode reward: 25.000000
episode reward: 33.000000
episode reward: 40.000000
episode reward: 71.000000
episode reward: 63.000000
episode reward: 53.000000
episode reward: 34.000000
episode reward: 60.000000
episode reward: 57.000000
episode reward: 50.000000
episode reward: 69.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 28.000000
episode reward: 14.000000
episode reward: 111.000000
episode reward: 55.000000
episode reward: 88.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 88.000000
episode reward: 101.000000
episode reward: 44.000000
episode reward: 137.000000
episode reward: 82.000000
episode reward: 25.000000
episode reward: 101.000000
episode reward: 48.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 120.000000
episode reward: 16.000000
episode reward: 36.000000
episode reward: 42.000000
episode reward: 68.000000
episode reward: 58.000000
episode reward: 85.000000
episode reward: 51.000000
episode reward: 89.000000
episode reward: 41.000000
episode reward: 18.000000
episode reward: 37.000000
episode reward: 22.000000
episode reward: 32.000000
episode reward: 61.000000
episode reward: 56.000000
episode reward: 45.000000
episode reward: 54.000000
episode reward: 40.000000
episode reward: 111.000000
episode reward: 153.000000
episode reward: 20.000000
episode reward: 40.000000
episode reward: 41.000000
episode reward: 83.000000
episode reward: 51.000000
episode reward: 85.000000
episode reward: 37.000000
episode reward: 12.000000
episode reward: 32.000000
episode reward: 138.000000
episode reward: 51.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 85.000000
episode reward: 74.000000
episode reward: 109.000000
episode reward: 95.000000
episode reward: 33.000000
episode reward: 113.000000
episode reward: 33.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 78.000000
episode reward: 162.000000
episode reward: 173.000000
episode reward: 14.000000
episode reward: 148.000000
episode reward: 11.000000
episode reward: 54.000000
episode reward: 108.000000
episode reward: 129.000000
episode reward: 52.000000
episode reward: 18.000000
episode reward: 89.000000
episode reward: 21.000000
episode reward: 88.000000
episode reward: 78.000000
episode reward: 93.000000
episode reward: 67.000000
episode reward: 158.000000
episode reward: 89.000000
episode reward: 53.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 93.000000
episode reward: 122.000000
episode reward: 33.000000
episode reward: 77.000000
episode reward: 29.000000
episode reward: 13.000000
episode reward: 102.000000
episode reward: 213.000000
episode reward: 31.000000
episode reward: 128.000000
episode reward: 60.000000
episode reward: 27.000000
episode reward: 15.000000
episode reward: 26.000000
episode reward: 49.000000
episode reward: 52.000000
episode reward: 33.000000
episode reward: 13.000000
episode reward: 60.000000
episode reward: 59.000000
episode reward: 125.000000
episode reward: 20.000000
episode reward: 103.000000
episode reward: 95.000000
episode reward: 65.000000
episode reward: 114.000000
episode reward: 17.000000
episode reward: 39.000000
episode reward: 198.000000
episode reward: 16.000000
episode reward: 163.000000
episode reward: 167.000000
episode reward: 85.000000
episode reward: 22.000000
episode reward: 39.000000
episode reward: 83.000000
episode reward: 49.000000
episode reward: 138.000000
episode reward: 111.000000
episode reward: 131.000000
episode reward: 89.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 69.000000
episode reward: 102.000000
episode reward: 107.000000
episode reward: 31.000000
episode reward: 47.000000
episode reward: 102.000000
episode reward: 101.000000
episode reward: 42.000000
episode reward: 155.000000
episode reward: 19.000000
episode reward: 131.000000
episode reward: 44.000000
episode reward: 18.000000
episode reward: 66.000000
episode reward: 16.000000
episode reward: 107.000000
episode reward: 120.000000
episode reward: 139.000000
episode reward: 116.000000
episode reward: 22.000000
episode reward: 138.000000
episode reward: 90.000000
episode reward: 66.000000
episode reward: 47.000000
episode reward: 42.000000
episode reward: 138.000000
episode reward: 72.000000
episode reward: 48.000000
episode reward: 69.000000
episode reward: 32.000000
episode reward: 27.000000
episode reward: 47.000000
episode reward: 45.000000
episode reward: 76.000000
episode reward: 125.000000
episode reward: 30.000000
episode reward: 91.000000
episode reward: 126.000000
episode reward: 179.000000
episode reward: 117.000000
episode reward: 50.000000
episode reward: 178.000000
episode reward: 71.000000
episode reward: 145.000000
episode reward: 52.000000
episode reward: 113.000000
episode reward: 83.000000
episode reward: 128.000000
episode reward: 116.000000
episode reward: 15.000000
episode reward: 61.000000
episode reward: 76.000000
episode reward: 27.000000
episode reward: 74.000000
episode reward: 33.000000
episode reward: 63.000000
episode reward: 13.000000
episode reward: 258.000000
episode reward: 123.000000
episode reward: 25.000000
episode reward: 118.000000
episode reward: 160.000000
episode reward: 22.000000
episode reward: 18.000000
episode reward: 76.000000
episode reward: 80.000000
episode reward: 80.000000
episode reward: 158.000000
episode reward: 72.000000
episode reward: 112.000000
episode reward: 103.000000
episode reward: 125.000000
episode reward: 57.000000
episode reward: 131.000000
episode reward: 125.000000
episode reward: 146.000000
episode reward: 136.000000
episode reward: 93.000000
episode reward: 26.000000
episode reward: 68.000000
episode reward: 142.000000
episode reward: 121.000000
episode reward: 101.000000
episode reward: 27.000000
episode reward: 75.000000
episode reward: 135.000000
episode reward: 47.000000
episode reward: 61.000000
episode reward: 41.000000
episode reward: 131.000000
episode reward: 104.000000
episode reward: 152.000000
episode reward: 140.000000
episode reward: 117.000000
episode reward: 138.000000
episode reward: 80.000000
episode reward: 75.000000
episode reward: 61.000000
episode reward: 15.000000
episode reward: 214.000000
episode reward: 170.000000
episode reward: 224.000000
episode reward: 81.000000
episode reward: 67.000000
episode reward: 155.000000
episode reward: 87.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 42.000000
episode reward: 158.000000
episode reward: 280.000000
episode reward: 189.000000
episode reward: 177.000000
episode reward: 75.000000
episode reward: 59.000000
episode reward: 26.000000
episode reward: 248.000000
episode reward: 163.000000
episode reward: 214.000000
episode reward: 61.000000
episode reward: 172.000000
episode reward: 13.000000
episode reward: 188.000000
episode reward: 119.000000
episode reward: 197.000000
episode reward: 164.000000
episode reward: 187.000000
episode reward: 276.000000
episode reward: 126.000000
episode reward: 157.000000
episode reward: 156.000000
episode reward: 129.000000
episode reward: 71.000000
episode reward: 144.000000
episode reward: 55.000000
episode reward: 114.000000
episode reward: 55.000000
episode reward: 111.000000
episode reward: 192.000000
episode reward: 77.000000
episode reward: 16.000000
episode reward: 163.000000
episode reward: 87.000000
episode reward: 45.000000
episode reward: 169.000000
episode reward: 146.000000
episode reward: 28.000000
episode reward: 192.000000
episode reward: 166.000000
episode reward: 23.000000
episode reward: 159.000000
episode reward: 129.000000
episode reward: 141.000000
episode reward: 272.000000
episode reward: 164.000000
episode reward: 229.000000
episode reward: 138.000000
episode reward: 232.000000
episode reward: 122.000000
episode reward: 126.000000
episode reward: 307.000000
episode reward: 124.000000
episode reward: 36.000000
episode reward: 204.000000
episode reward: 56.000000
episode reward: 113.000000
episode reward: 131.000000
episode reward: 116.000000
episode reward: 101.000000
episode reward: 187.000000
episode reward: 56.000000
episode reward: 295.000000
episode reward: 327.000000
episode reward: 57.000000
episode reward: 198.000000
episode reward: 55.000000
episode reward: 90.000000
episode reward: 102.000000
episode reward: 31.000000
episode reward: 102.000000
episode reward: 107.000000
episode reward: 130.000000
episode reward: 367.000000
episode reward: 60.000000
episode reward: 47.000000
episode reward: 212.000000
episode reward: 261.000000
episode reward: 293.000000
episode reward: 63.000000
episode reward: 107.000000
episode reward: 85.000000
episode reward: 226.000000
episode reward: 190.000000
episode reward: 176.000000
episode reward: 220.000000
episode reward: 186.000000
episode reward: 90.000000
episode reward: 206.000000
episode reward: 147.000000
episode reward: 182.000000
episode reward: 177.000000
episode reward: 181.000000
episode reward: 156.000000
episode reward: 61.000000
episode reward: 137.000000
episode reward: 159.000000
episode reward: 114.000000
episode reward: 190.000000
episode reward: 89.000000
episode reward: 21.000000
episode reward: 46.000000
episode reward: 138.000000
episode reward: 166.000000
episode reward: 122.000000
episode reward: 161.000000
episode reward: 108.000000
episode reward: 145.000000
episode reward: 125.000000
episode reward: 38.000000
episode reward: 125.000000
episode reward: 130.000000
episode reward: 128.000000
episode reward: 165.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 56.000000
episode reward: 129.000000
episode reward: 134.000000
episode reward: 126.000000
episode reward: 137.000000
episode reward: 135.000000
episode reward: 144.000000
episode reward: 219.000000
episode reward: 70.000000
episode reward: 115.000000
episode reward: 130.000000
episode reward: 213.000000
episode reward: 148.000000
episode reward: 115.000000
episode reward: 129.000000
episode reward: 148.000000
episode reward: 137.000000
episode reward: 187.000000
episode reward: 128.000000
episode reward: 140.000000
episode reward: 139.000000
episode reward: 164.000000
episode reward: 150.000000
episode reward: 153.000000
episode reward: 119.000000
episode reward: 32.000000
episode reward: 159.000000
episode reward: 163.000000
episode reward: 167.000000
episode reward: 85.000000
episode reward: 49.000000
episode reward: 149.000000
episode reward: 180.000000
episode reward: 151.000000
episode reward: 147.000000
episode reward: 141.000000
episode reward: 198.000000
episode reward: 134.000000
episode reward: 265.000000
episode reward: 147.000000
episode reward: 176.000000
episode reward: 126.000000
episode reward: 165.000000
episode reward: 163.000000
episode reward: 236.000000
episode reward: 175.000000
episode reward: 139.000000
episode reward: 119.000000
episode reward: 225.000000
episode reward: 222.000000
episode reward: 203.000000
episode reward: 155.000000
episode reward: 134.000000
episode reward: 93.000000
episode reward: 181.000000
episode reward: 147.000000
episode reward: 139.000000
episode reward: 158.000000
episode reward: 131.000000
episode reward: 137.000000
episode reward: 167.000000
episode reward: 140.000000
episode reward: 122.000000
episode reward: 130.000000
episode reward: 142.000000
episode reward: 161.000000
episode reward: 122.000000
episode reward: 180.000000
episode reward: 101.000000
episode reward: 105.000000
episode reward: 146.000000
episode reward: 129.000000
episode reward: 144.000000
episode reward: 107.000000
episode reward: 126.000000
episode reward: 107.000000
episode reward: 138.000000
episode reward: 121.000000
episode reward: 43.000000
episode reward: 125.000000
episode reward: 108.000000
episode reward: 33.000000
episode reward: 114.000000
episode reward: 126.000000
episode reward: 119.000000
episode reward: 103.000000
episode reward: 111.000000
episode reward: 142.000000
episode reward: 105.000000
episode reward: 131.000000
episode reward: 111.000000
episode reward: 20.000000
episode reward: 116.000000
episode reward: 118.000000
episode reward: 101.000000
episode reward: 103.000000
episode reward: 97.000000
episode reward: 15.000000
episode reward: 94.000000
episode reward: 63.000000
episode reward: 107.000000
episode reward: 51.000000
episode reward: 19.000000
episode reward: 106.000000
episode reward: 51.000000
episode reward: 21.000000
episode reward: 57.000000
episode reward: 39.000000
episode reward: 108.000000
episode reward: 132.000000
episode reward: 25.000000
episode reward: 26.000000
episode reward: 76.000000
episode reward: 59.000000
episode reward: 46.000000
episode reward: 47.000000
episode reward: 39.000000
episode reward: 32.000000
episode reward: 100.000000
episode reward: 62.000000
episode reward: 109.000000
episode reward: 117.000000
episode reward: 61.000000
episode reward: 122.000000
episode reward: 111.000000
episode reward: 106.000000
episode reward: 105.000000
episode reward: 129.000000
episode reward: 26.000000
episode reward: 78.000000
episode reward: 127.000000
episode reward: 105.000000
episode reward: 117.000000
episode reward: 116.000000
episode reward: 103.000000
episode reward: 151.000000
episode reward: 107.000000
episode reward: 139.000000
episode reward: 132.000000
episode reward: 122.000000
episode reward: 113.000000
episode reward: 128.000000
episode reward: 85.000000
episode reward: 121.000000
episode reward: 172.000000
episode reward: 91.000000
episode reward: 123.000000
episode reward: 116.000000
episode reward: 127.000000
episode reward: 143.000000
episode reward: 116.000000
episode reward: 164.000000
episode reward: 116.000000
episode reward: 131.000000
episode reward: 125.000000
episode reward: 115.000000
episode reward: 169.000000
episode reward: 123.000000
episode reward: 128.000000
episode reward: 178.000000
episode reward: 153.000000
episode reward: 78.000000
episode reward: 134.000000
episode reward: 165.000000
episode reward: 124.000000
episode reward: 79.000000
episode reward: 132.000000
episode reward: 128.000000
episode reward: 153.000000
episode reward: 108.000000
episode reward: 126.000000
episode reward: 159.000000
episode reward: 117.000000
episode reward: 131.000000
episode reward: 19.000000
episode reward: 119.000000
episode reward: 145.000000
episode reward: 135.000000
episode reward: 132.000000
episode reward: 112.000000
episode reward: 141.000000
episode reward: 27.000000
episode reward: 124.000000
episode reward: 125.000000
episode reward: 144.000000
episode reward: 109.000000
episode reward: 127.000000
episode reward: 29.000000
episode reward: 131.000000
episode reward: 34.000000
episode reward: 136.000000
episode reward: 127.000000
episode reward: 141.000000
episode reward: 119.000000
episode reward: 139.000000
episode reward: 117.000000
episode reward: 121.000000
episode reward: 52.000000
episode reward: 137.000000
episode reward: 140.000000
episode reward: 153.000000
episode reward: 141.000000
episode reward: 162.000000
episode reward: 141.000000
episode reward: 134.000000
episode reward: 61.000000
episode reward: 127.000000
episode reward: 23.000000
episode reward: 176.000000
episode reward: 174.000000
episode reward: 182.000000
episode reward: 166.000000
episode reward: 133.000000
episode reward: 174.000000
episode reward: 184.000000
episode reward: 192.000000
episode reward: 165.000000
episode reward: 202.000000
episode reward: 68.000000
episode reward: 99.000000
episode reward: 229.000000
episode reward: 236.000000
episode reward: 244.000000
episode reward: 239.000000
episode reward: 195.000000
episode reward: 224.000000
episode reward: 316.000000
episode reward: 58.000000
episode reward: 274.000000
episode reward: 314.000000
episode reward: 228.000000
episode reward: 300.000000
episode reward: 247.000000
episode reward: 395.000000
episode reward: 101.000000
episode reward: 220.000000
episode reward: 355.000000
episode reward: 422.000000
episode reward: 282.000000
episode reward: 241.000000
episode reward: 257.000000
episode reward: 198.000000
episode reward: 397.000000
episode reward: 220.000000
episode reward: 188.000000
episode reward: 252.000000
episode reward: 96.000000
episode reward: 303.000000
episode reward: 278.000000
episode reward: 227.000000
episode reward: 98.000000
episode reward: 187.000000
episode reward: 249.000000
episode reward: 199.000000
episode reward: 208.000000
episode reward: 218.000000
episode reward: 209.000000
episode reward: 111.000000
episode reward: 203.000000
episode reward: 184.000000
episode reward: 81.000000
episode reward: 199.000000
episode reward: 253.000000
episode reward: 218.000000
episode reward: 230.000000
episode reward: 147.000000
episode reward: 193.000000
episode reward: 184.000000
episode reward: 191.000000
episode reward: 206.000000
episode reward: 71.000000
episode reward: 175.000000
episode reward: 190.000000
episode reward: 216.000000
episode reward: 139.000000
episode reward: 188.000000
episode reward: 157.000000
episode reward: 183.000000
episode reward: 66.000000
episode reward: 153.000000
episode reward: 159.000000
episode reward: 143.000000
episode reward: 195.000000
episode reward: 137.000000
episode reward: 156.000000
episode reward: 104.000000
episode reward: 130.000000
episode reward: 132.000000
episode reward: 154.000000
episode reward: 150.000000
episode reward: 170.000000
episode reward: 47.000000
episode reward: 152.000000
episode reward: 66.000000
episode reward: 140.000000
episode reward: 139.000000
episode reward: 192.000000
episode reward: 166.000000
episode reward: 142.000000
episode reward: 106.000000
episode reward: 142.000000
episode reward: 132.000000
episode reward: 153.000000
episode reward: 89.000000
episode reward: 190.000000
episode reward: 145.000000
episode reward: 25.000000
episode reward: 138.000000
episode reward: 29.000000
episode reward: 140.000000
episode reward: 135.000000
episode reward: 149.000000
episode reward: 200.000000
episode reward: 169.000000
episode reward: 164.000000
episode reward: 144.000000
episode reward: 145.000000
episode reward: 169.000000
episode reward: 213.000000
episode reward: 172.000000
episode reward: 180.000000
episode reward: 187.000000
episode reward: 196.000000
episode reward: 214.000000
episode reward: 177.000000
episode reward: 176.000000
episode reward: 180.000000
episode reward: 186.000000
episode reward: 207.000000
episode reward: 221.000000
episode reward: 253.000000
episode reward: 153.000000
episode reward: 196.000000
episode reward: 48.000000
episode reward: 218.000000
episode reward: 194.000000
episode reward: 269.000000
episode reward: 269.000000
episode reward: 276.000000
episode reward: 194.000000
episode reward: 271.000000
episode reward: 128.000000
episode reward: 126.000000
episode reward: 239.000000
episode reward: 273.000000
episode reward: 274.000000
episode reward: 236.000000
episode reward: 236.000000
episode reward: 303.000000
episode reward: 32.000000
episode reward: 241.000000
episode reward: 260.000000
episode reward: 251.000000
episode reward: 338.000000
episode reward: 292.000000
episode reward: 244.000000
episode reward: 392.000000
episode reward: 211.000000
episode reward: 329.000000
episode reward: 282.000000
episode reward: 333.000000
episode reward: 11.000000
episode reward: 302.000000
episode reward: 344.000000
episode reward: 161.000000
episode reward: 358.000000
episode reward: 464.000000
episode reward: 343.000000
episode reward: 525.000000
episode reward: 351.000000
episode reward: 602.000000
episode reward: 230.000000
episode reward: 71.000000
episode reward: 450.000000
episode reward: 193.000000
episode reward: 90.000000
episode reward: 787.000000
episode reward: 668.000000
episode reward: 1249.000000
episode reward: 32.000000
episode reward: 360.000000
episode reward: 640.000000
episode reward: 756.000000
episode reward: 1204.000000
episode reward: 396.000000
episode reward: 124.000000
episode reward: 1434.000000
episode reward: 521.000000
episode reward: 185.000000
episode reward: 532.000000
ran 2402 episodes, avg reward: 62.447960

















In [28]:



    


# plot the episode rewards (balancing steps)
%matplotlib inline
import matplotlib.pyplot as plt
plt.xlabel('episode')
plt.ylabel('balancing steps')
plt.bar(np.arange(len(rew)),rew)



    


















    
Out[28]:








<Container object of 2402 artists>

Assignment

  1. Adapt the create_mlp function to try different network architectures. You can experiment with different depths (more layers) , different sizes (more nodes/layer) or regularization (weight decay/dropout).
  2. Try different combinations of learning rate (alpha) and discount (gamma). Plot the results to see which gives the best results.
  3. (Advanced - optional): Try improving the learning agent's performance by adding a critic. A critic evaluates the learning agent by estimating the Q-values of the current policy. You can train a critic by training a second network to predict Q-values. This network takes as inputs the observations and outputs Q-values. You can train it using the same observations and returns as used in standard PG with target $\frac{1}{T}\sum_{t=0}^{T}(Q_\omega(s_t,a_t) - R_t)^2$ where $Q_\omega(s_t,a_t)$ is the output Q-value predicted by the network for the action that was taken at time $t$ and $R_t$ is the return achieved (same as in the cost function above). The policy gradient target then becomes $ - \frac{1}{T}\sum_{t=0}^{T} log \pi_\theta(s_t,a_t) Q_\omega(s_t,a_t)$ (replace R with Q) where $\pi_\theta(s_t,a_T)$ is the action probability output by the first network and $Q_\omega(s_t,a_t)$ are the action Q-values predicted by the second network. In practice, people often use a single network with 2 output layers, 1 for probabilities and 1 for Q-values. More details: http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching_files/pg.pdf




In [ ]:

Content source: vub-ai-lab/MAL-seminar

Similar notebooks:

notebook.community | gallery | about